An immersive multisensory virtual reality approach to the study of human-built environment interactions: Technical workflows

Virtual Reality technology has gained increased attention due to its capacity to provide immersive and interactive experiences to its users. Although increasing evidence has suggested that incorporating multisensory components in VR can promote the sense of presence and improve user performance, most of the current VR applications are limited to visual and auditory senses. In this article, a novel method of integrating thermal-related devices (heat lamps and fans) into Virtual Reality was developed. Automated interaction with the thermal-related devices was achieved using Arduino-based control module with its program embedded into the VR platform-Unity. The functions, hardware and software requirements of the multisensory Virtual Reality system as well as the step-by-step procedures are detailed to provide a reproducible workflow for future applications.• A practical workflow to integrate thermal apparatus into Virtual Reality.• Dynamic airflow and radiative heating incorporated into Virtual Reality.• Automated process to allow user interaction with the thermal components in Virtual Reality.


Introduction
Virtual Reality (VR) technology offers great potential for applications in the field of Architecture, Engineering and Construction as a research method to study human-building interaction [ 7 , 9 ], as a pre-occupancy evaluation tool [18] or as a client engagement tool [5] . However, most of the current VR applications have limited capacity, simulating only the visual and auditory environment. Integrating multisensory components (e.g., smell, hot/cold, wind, radiation) into Virtual Reality can improve the sense of presence, the feeling of being actually there in the virtual environment [15] , and user performance [6] . Adding thermal sense to Virtual Reality can open new opportunities. For example, research questions involving human thermal comfort and thermal adaptive behavior in built environments can be studied with this method for the capacity to conduct laboratory-controlled experiment with enhanced sense of immersion and realism to maximize the ecological validity [ 1 , 16 ]. The conventional way of presenting thermal comfort simulation results to stakeholders, which usually relies on graphics and abstract numbers, can be radically transformed by integrating the ability to experientially simulate the embedded multisensory qualities of the design proposal. Various recent studies have explored the possibilities of integrating thermo-haptic feedback into VR, for example, a liquid metal based multimodal sensor and haptic feedback device for thermal and tactile sensation generation in Virtual Reality [13] , a stretchable copper nanowire-based heater for replication of the feeling of heat in a virtual world [8] . Besides the thermal hot feeling, recent progresses in VR thermo-haptic demonstrated cold feeling in VR using stretchable skin-like thermoelectric devices [ 10 , 11 , 14 ].
This project aims to provide a practical and reproducible workflow for integrating thermal sense into Virtual Reality. The potential applications of the Multisensory Immersive Virtual Environment (MIVE) has been illustrated in [12] . The multisensory immersive virtual reality system incorporates the simulation of dynamic airflow and solar radiation into VR and enables the user to effortlessly interact with the thermal environment during their VR experience.

System function and composition
This section describes the function, working mechanism, hardware, and software requirements of the multisensory immersive VR system.

Function
The multisensory immersive VR system enables the user to perceive a dynamic simulation of sunlight radiation and natural wind from a range of sensory output devices including fans and halogen heat lamps. Moreover, the operation of these sensory output devices can be modified by the VR users, either by navigating to different microclimatic areas in the virtual environment or by modifications of the virtual thermal environment directly in the VR.

Working mechanism
The multisensory VR system is composed of sensory output devices (VR headset, halogen heat lamps and fans), control modules made from Arduino components [2] and integrated development environment of Unity3D (2022), see Fig. 1 .
The intensities of thermal stimuli can be simulated by digitally control the operation of the halogen heat lamps and fans. The frame that mounts the heat lamps is connected to its stand through adjustable hinge joints, allowing users to modify the altitude (elevation) angle of the radiant heat source. This adjustment facilitates the simulation of changing solar angles throughout the year. Users have the freedom to change the relative azimuth angle of the heat lamps by physically orienting themselves in different directions. By defining the relative orientation of the VR avatar to the VR scene and rendering appropriate lighting, it becomes possible to simulate different sunlight angles within a day. Currently, the system utilizes only two fans set in the same direction to simulate wind. 1 However, additional fans can be deployed in different directions to provide a more comprehensive simulation of wind from various directions.
Cable protector mats are used to tape-attaching all the exposed cables to the floor in case of possible trapping hazard. Moreover, a playable area (virtual boundary) is defined when starting the VR experience to prevent users from hitting any devices.
During the experience in VR, thermal-related actions initiated by the users (e.g., moving to shade from sunlight) trigger virtual event function in Unity's runtime system based on C# language. Signals are sent to the pre-programed control modules to activate certain functions for the operation of the fan or heat lamps ( Fig. 2 ).

Workflow
The process of setting up multisensory VR system can be divided into three steps: hardware set-up, programming Arduino, Unity Integration . This section describes the practical workflow for establishing the thermally integrated VR system. The process of determining the simulated thermal condition is explained in Lyu et al. [12] .

Dynamic airflow simulation in VR
The aim of the dynamic airflow simulation is to: (1) simulate the turbulence characteristics of a time-series recording of actual wind. (2) allow users to have the ability to interact with different airflow profiles during VR experience, e.g., choose different wind profiles or modify wind speed. The fluctuation of air velocity is achieved by transforming the air velocity values into a sequence of control signals for the fans to operate at the respective levels ( Table 1 ).

Hardware and software components
Step one: hardware set-up The hardware set-up of the dynamic airflow simulation consists of the assembly of the Wind Control Module, fans, and VR headset. The Wind Control Module is made of Arduino UNO Rev3 (microcontroller), solderless breadboard (circuit prototyping board), infrared LED receiver, infrared transmitting LED and resistor. It works as a digitally programed remote control for the operation of the fan to simulate the turbulence characteristics of natural wind. Two functions are achieved, the ability to receive IR signal from the fan's original remote controller to configure the Arduino Uno Rev3, and the ability to send IR signal to control the fan. Firstly, a series circuit was designed to connect the infrared LED receiver, infrared transmitting LED and the Arduino Rev 3 microcontroller with the Arduino jumper wires ( Fig. 3 ). Wind Control Module can be connected to the PC from its USB connector so that it can get powered (5v) and receive signal from the serial port in the PC. The infrared transmitting LED has a limited viewing angle (normally 30°-40°), so during operation it should point towards the location of IR sensor on the fan ( Fig. 4 ).
Step two: programming Arduino The Arduino Uno Rev3 was programmed in the Arduino IDE (based on C ++ ) with two scripts, IR signal receiver and IR signal sender (see supplement files). IR signal receiver was written to receive signal from the IR LED receiver (pin 8 on Arduino Uno Rev3) while the IR signal sender to send signal via the Infrared Transmitting LED to the fans.    Step three: calibrate the fan and generate airflow commands (1) Determine fan's location in relation to the user.
The relative position of the fan in relation to the VR user should be calibrated with an anemometer, matching the air velocity measured at the fan's highest and lowest operating levels with the maximum and minimum target airflow velocity (m/s). Note: Use the maximum air velocity to determine the distance of the fan to the user. If the minimum air velocity cannot be matched with the air velocity at the fan's lowest operating level, more fans are needed.
(2) Generate airflow profile commands The target airflow velocity should be converted to discrete air velocity levels, responding to different fan operating levels. The previous signal commands for increase and decrease fan speed were given variable names in the C# script in Unity: Inc () and Dec () . The discreate air velocity level sequence can be converted to a series of commands with Inc () and Dec (). For example, if air velocity level change from level 1 to 4 in a second, three Dec () commands will be used. Note: variable Inc () and Dec () will be defined in the C# script in step four.
Step four: integration into Unity C# script can be executed during Unity VR runtime. A function Wind Script (see supplement files) in C# was created to be able to send commands to the programed Arduino UNO Rev3 directly from Unity (instead of from Arduino IDE). In this way, the commands for controlling the fan to change operating levels in order to simulate target airflow can be embedded in Unity's platform and this function can be called from other actions in VR. For example, Inc() and Dec() were defined to speed up and down the fan's operating levels. Note: In this case, if the user enters a wind zone in the VR environment, defined as the collision between the GameObject of the player and the GameObject of the wind zone, an event will be triggered to active the wind profile function in wind script . If the user exits that zone, CloseFan function will be executed. OnTriggerEnter and OnTriggerExit (Unity technology, 2022) were used to achieve this .
Step-by-step procedures are listed below.

Virtual sun simulation in VR
Simulation of the virtual sun in VR is achieved by using a bank of 10 quartz tungsten halogen lamps (250 W × 10) mounted on a 0.6 m × 0.9 m frame. Radiative heat from the halogen lamp offers good approximation to that of the solar radiation because of its close match to the infrared waveband in the solar spectrum [ 17 , 19 ]. The operation of halogen lamps (ON/OFF) is controlled by the Sunlight Control Module for user interaction in the VR environment. Comprising of two sets of Arduino microcontrollers (Arduino UNO Rev3), prototyping board shields and power relays (DuinoTECH XC-4419), the Sunlight Control Module can receive signals from the triggered events in Unity and control the status of the heat lamps to simulate a dynamic experience of sunlight ( Table 2 ).

Hardware and software components
Step one: hardware set-up The Sunlight Control Module works as a digitally controlled power relay for the operation of the halogen lamps. Instead of using the Solderless Breadboard, prototyping shields were used to provide a permanent configuration of the circuit. In addition, the control module was mounted into a sealed polycarbonate box which has openings for connections to computer and power source. Therefore, the need for physical access to the circuit can be minimized for electrical safety (as direct current is controlled via this module). No resistor is added in the circuit because the power relay contains one. The capacity of the power relay (10Amp 250 Volts, capable of supporting 2500 W external device) to control the halogen lamps was checked ( Figs. 5 and 6 ).
(1) Connect the power relay to the Arduino prototyping board shield with soldering. Note: Connect S to any of the digital pin in the prototyping shield (in the example, pin 12 was used), "+ " to the 5 V pin, and "-" to the Ground pin. Radiation flux density may be measured for this purpose.  Step two: programming Arduino Programming for the Sunlight Control Module consists of the functionality for receiving serial commands from the USB port in PC with the Serial.readBytesUntil [4] function and activate specific halogen lamp operation definitions. Note: Halogen lamp operation mode is defined using digitalWrite [3] which writes a HIGH or a LOW value to a digital pin connecting the power relay (HIGH will cause power relay to open the circuit whereas LOW will cause it to close the circuit). For example, when receiving a character value of "O ", digitalWrite will output HIGH ( 1 in digital signal) value to open the circuit for five halogen lamps. The script Heatlamp-arduino-script (see supplement files) needs to be uploaded to Arduino via Arduino IDE.
Step three: integration into Unity The hardware configuration can simulate three sunlight conditions: shaded (all halogen lamps off), partially shaded (five halogen lamps off) and sunlit (all halogen lamps on). These are defined as three functions in the script to control the operation of the halogen lamps by sending specific serial command to the Sunlight Control Modules , e.g., sending serial commands "C " to turn off power connection to two sets of halogen heat lamps. These functions can be called by event trigger in Unity, either by Unity's User Interface or by user enter/exit certain areas.
(1) Create an empty GameObject named Sunlight . Attach the script Artificial Sun Script onto this GameObject, fill in port names for the USB port used, e.g., COM1. Note: the port names can be found in Device Manager function in Windows operating system.

Limitation and recommendation
This research develops a method of integrating immersive thermal experience into Virtual Reality with airflow and radiation simulation. The multisensory VR system is capable of simulating the outdoor dynamic thermal experience and affording user interaction with the thermal environment via the Arduino-based control modules. Limitations that demand future development are related to (1) the Arduino control module and (2) the sensory output devices: The Arduino control module for the virtual sun uses a power relay to control the ON/OFF status of the halogen lamps which is only a binary operation. Control of voltage for the power supply to achieve a continuous change of radiation from the halogen lamp can be future investigated, potentially with pulse-width modulation (PWM) method. Furthermore, only halogen lamps and bladeless fans were used in the prototypes. More sensory output devices, such as mist fans for humidity delivery, conductive heating or cooling devices and scent delivery devices, can be tested with the same principle described in this article.

Ethics statements
The relevant informed consent was obtained from subjects participated in this study.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability
No data was used for the research described in the article.